WWC Single Study Review 

U.S. DEPARTMENT OF EDUCATION 

What Works Clear inghouse™ 

• 

1 P ^ INSTITUTE of 

■ W 4# EDUCATION SCIENCES 

July 2013 


WWC Review of the Report “Information and College Access: 
Evidence from a Randomized Field Experiment ” 1 


The findings from this review do not reflect the full body of research evidence on the effects of 
providing high school students with information about college costs and returns on college enrollment. 


What is this study about? 

The study examined the impact of offering financial 
aid informational materials on the postsecondary 
expectations of high school students. 

Approximately 5,000 students from five low-achieving 
high schools in Ontario, Canada were invited to 
complete an online survey about their postsecondary 
aspirations. About 1 ,600 students did so, and after 
completing the survey, were randomly assigned to 
one of two groups. 

Students in the intervention group were first shown 
an online 3-minute video about the costs and typi- 
cal yearly earnings associated with postsecondary 
education. Next, students were invited to try out a 
financial aid calculator to help estimate their college 
costs. Finally, students were given the financial aid 
application package for local postsecondary institu- 
tions, along with instructions on how to apply for aid. 

Students in the comparison group did not receive 
any of the informational materials described above. 

The study measured the impact of offering these 
materials on student postsecondary expectations 
3 weeks after the intervention. 2 


Features of Providing Information 
About College Costs and Returns 


The authors of this study designed an intervention 
to address their belief that low-income students 
often underestimate the economic value and 
overestimate the costs of postsecondary education. 
The intervention had three parts, all of which were 
delivered via the Internet: 

• a short video that was designed to convey 
information about typical earnings at different 
levels of educational attainment, the expected 
costs of postsecondary education, and financial 
aid eligibility; 

• a financial aid calculator that allowed students to 
estimate the value of grant and loan funding for 
which they might be eligible; and 

• the financial aid application packages for local 
postsecondary institutions, along with instructions 
on how to apply for aid. 
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What did the study find? 

The authors reported, and the WWC confirmed, a 
statistically significant impact of offering the infor- 
mational materials on the educational expectations 
of the high school students in the study, with a 
larger percentage of students in the intervention 
group expecting to attend postsecondary educa- 
tion (88%) than in the comparison group (82%). The 
strongest effects were observed 3 weeks after the 
intervention for students who reported being unsure 
about their postsecondary plans before the inter- 
vention (64% in the intervention group vs. 42% in 
the comparison group). There was also a statistically 
significant difference between the intervention and 
comparison groups for students who were initially 
certain that they would attend a postsecondary 
institution (93% in the intervention group reported 
that they would attend a postsecondary institution 
during the second survey, vs. 89% in the compari- 
son group). 


WWC Rating 


The research described in this 
report meets WWC evidence 
standards without reservations 

Strengths: This study is a well-implemented 
randomized controlled trial. 

Cautions: This study measured postsecondary 
enrollment expectations, not actual enrollment. 
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Appendix A: Study details 

Oreopoulos, P., & Dunn, R. (2012). Information and college access: Evidence from a randomized field 
experiment (NBER Working Paper No. 18551). Retrieved from http://www.nber.org/papers/w18551 


Setting 

The study was conducted in five low-achieving public high schools located in low-income 
neighborhoods in Toronto, Canada. All five high schools were in the bottom tenth percentile 
of schools in Ontario in terms of average student achievement. The study authors assert that 
typically, about 30% of the students in these five high schools attend a postsecondary institu- 
tion immediately after graduation. 

Study sample 

All students in the five high schools were invited to participate (n = 5,017) and were offered 
a $20 honorarium for completing two online surveys. The analytic sample was comprised of 
1,616 students in grades 9-12 who completed both surveys and had been randomly assigned 
to either the intervention group (n = 81 1) or the comparison group (n = 805). About 52% of 
students in the analysis sample were female. Prior to the intervention, 12% of the students 
reported being unsure about their postsecondary enrollment plans. 

Intervention 

group 

Students in the intervention group were first shown an online 3-minute video about the costs 
and typical yearly earnings associated with postsecondary education. Next, students were 
invited to try out an online financial aid calculator to help estimate their college costs. Finally, 
students were given an online financial aid application package for local postsecondary insti- 
tutions, along with instructions on how to apply for aid. The typical intervention group student 
spent about three minutes viewing the informational materials. 

Comparison 

group 

Students in the comparison group did not receive the informational materials that were pro- 
vided to the intervention group. 

Outcomes and 
measurement 

Three weeks after completing the first survey (and, for the students in the intervention group, 
viewing the informational materials), study authors measured students’ educational expecta- 
tions by administering a second survey. For a more detailed description of the survey question 
used, see Appendix B. 

Support for 
implementation 

The study authors did not provide detail on the development of the online materials or on any 
needed support (e.g., technical support). The study authors did provide evidence that the 
intervention was implemented relatively well. They reported that the median amount of time 
spent on the web page was highly similar to the length of the video, suggesting that the typical 
student played much of the video. 

Reason for 
review 

This study was identified for review by the WWC by receiving significant media attention. 
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Appendix B: Outcome measure for the postsecondary enrollment domain 


Postsecondary enrollment 


Educational expectations 

This outcome is based on a single survey question that asks students to report whether they expect not to 
pursue postsecondary education, whether they expect to attain a community college (i.e., 2-year) degree, or 
whether they expect to attain a university (i.e., 4-year) degree. For this analysis, the study authors estimated 
the percentage of students in the intervention and comparison groups who reported expecting to attain any 
postsecondary degree. 
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Appendix C: Study findings for the postsecondary enrollment domain 





Mean 

(standard deviation) 


WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Postsecondary enrollment 

Educational expectations 

Full sample 

5 schools/ 
975 students 

0.88 

(0.32) 

0.82 

(0.38) 

6% 

0.16 

+6 

<0.05 

Domain average for postsecondary enrollment 




0.16 

+6 

Statistically 

significant 


Table Notes: Positive results for mean difference, effect size, and improvement index favor the intervention group; negative results favor the comparison group. The effect size is 
a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) in an average student’s outcome that can 
be expected if the student is given the intervention. The improvement index is an alternate presentation of the effect size, reflecting the change in an average student's percentile 
rank that can be expected if the student is given the intervention. 

Study Notes: No corrections for clustering or multiple comparisons were needed. The data are based on means and standard errors obtained from the study authors. The p-value 
presented here was computed by the WWC. The study authors applied statistical controls for student gender, whether the student was born in Canada, last year’s self-reported 
grades, parental education, whether the student had thought about dropping out of school, whether the student believed that the Canadian government guarantees postsecondary 
access, and expected highest degree earned (pre-intervention). The group means and the mean difference are adjusted for the variables listed above but the standard deviations 
are not. Please see the WWC Handbook version 2.1 for more information. The standard deviations are based on robust standard errors (that correct for clustering at the high 
school level) provided by the study authors. 
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Appendix D: Supplemental findings by domain 





Mean 

(standard deviation) 

WWC calculations 

Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention Comparison 
group group 

Mean Effect Improvement 

difference size index p-value 


Postsecondary enrollment 


Educational expectations 

Unsure about 
educational 
attainment 
(pre-intervention) 

5 schools/ 
101 students 

0.64 

(0.48) 

0.42 

(0.49) 

22% 

0.45 

+17 

<0.05 

Educational expectations 

Expecting not to 
complete more 
than high school 
(pre-intervention) 

5 schools/ 
25 students 

0.63 

(0.48) 

0.46 

(0.50) 

17% 

0.31 

+12 

0.45 

Educational expectations 

Expecting to 
complete college 
or university 
(pre-intervention) 

5 schools/ 
849 students 

0.93 

(0.25) 

0.89 

(0.31) 

4% 

0.15 

+6 

<0.05 


Table Notes: Positive results for mean difference, effect size, and improvement index favor the intervention group; negative results favor the comparison group. The effect size is 
a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) in an average student’s outcome that can 
be expected if the student is given the intervention. The improvement index is an alternate presentation of the effect size, reflecting the change in an average student's percentile 
rank that can be expected if the student is given the intervention. 

Study Notes: A correction for clustering was not needed. A correction for multiple comparisons was needed but did not affect significance levels; therefore, the p-values presented 
here are the values that were reported in the original study. The sample sizes were provided by the study authors. The study authors applied statistical controls for student gender, 
whether the student was born in Canada, last year's self-reported grades, parental education, whether the student had thought about dropping out of school, whether the student 
believed that the Canadian government guarantees postsecondary access, and expected highest degree earned (pre-intervention). The group means and the mean differences are 
adjusted for the variables listed above but the standard deviations are not. Please see the WWC Handbook version 2.1 for more information. The standard deviations are based on 
robust standard errors (that correct for clustering at the high school level) given in the report. 

The analysis of educational aspirations for the group that was unsure about educational attainment (pre-intervention) experienced a relatively high degree of attrition (47% overall, 
43% in the intervention group, and 53% in the comparison group). The report provides evidence that the analytic samples were comparable in terms of their academic achievement 
(and several other variables). For the other analyses, the amount of attrition was relatively low. 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors]) to assess whether the study design meets WWC evidence standards. The review reports the WWC’s assessment of whether 
the study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the Postsecondary Education topic area review protocol, version 2.0. A quick review of 
this study was released on January 8, 2013, and this report is the follow-up review that replaces that initial assessment. The WWC 
rating applies only to the results that were eligible under this topic area and met WWC standards without reservations or met WWC 
standards with reservations, and not necessarily to all results presented in the study (see Note 2 below). 

2 The study authors measured several other outcomes. These were: (a) expected earnings at age 35 for high school graduates, com- 
munity college graduates, and university graduates; (b) the ratio of college to high school expected earnings and university to college 
expected earnings; (c) several measures of financial aid expectations; (d) whether students sought additional information; and (e) 
expected end-of-year grades. These outcomes were not included in this report because they are not identified as eligible outcomes in 
the Postsecondary Education topic area protocol. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2013, July). WWC 
review of the report: Information and college access: Evidence from a randomized field experiment. Retrieved 
from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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